Method, system and software product for optimizing the delivery of content to a candidate

ABSTRACT

A computer-implemented method and system for improving recipient response rates to content presented via a data communications network to a plurality of recipients based upon recipient response data, wherein the content comprises a plurality of content factors. Each content factor is selected from one or more corresponding content levels. A data structure is provided for containing recipient response data associated with content comprising combinations of content levels. A plurality of content samples is generated by forming combinations of content levels for a plurality of content factors in accordance with respective combination weightings derived from recipient response data contained within the data structure. The plurality of content samples is presented to a corresponding plurality of recipients via the data communications network. The recipient responses to the content samples are received via the data communications network. The recipient responses are analysed to identify one or more combinations of content levels resulting in high recipient response rates relative to other combinations in accordance with predetermined statistical criteria. The recipient response data associated with combinations of content levels resulting in high recipient response rates is stored within the data structure.

FIELD OF THE INVENTION

The present invention relates generally to computerised informationsystems. In particular, the invention concerns a method and system forimproving response rate to online content, based upon candidate responsedata and content parameters. The invention is most readily implementedon the Internet and it will therefore be convenient to describe theinvention in that environment. It should be understood however that theinvention may be implemented in other environments.

BACKGROUND TO THE INVENTION

Online media offer a number of advantages over traditional advertisingdistribution techniques; most importantly, the speed of execution anddelivery, and the verifiability of content performance. However, thetechnical sophistication of marketing techniques has lagged behind thatof the distribution and consumption mechanisms upon which they rely, andmany marketers have yet to fully realise the unique benefits of theonline space.

The increasing prominence of web analytics and optimisation solutions inthe suite of tools offered to online marketers is a sign that techniquesare becoming more sophisticated, and that the marketing community isdiscovering the improvements in strategy and execution attainable viaconstant measurement and experimentation.

The performance of online marketing campaigns (including a coordinatedset of email broadcasts, website landing pages, or banneradvertisements) is often tracked in terms of click-through rates (theproportion of recipients of a particular content item who click on it),and conversion rates (the proportion of recipients of a particularcontent item who go on to complete a target action, such as purchasing aproduct). In order to improve these metrics, a number of optimisationprocesses are used, such as A/B testing or multivariate testing.However, these optimisation methods are labour-intensive and require aconsiderable amount of data to be processed over the course of amarketing campaign. In addition, campaigns consisting of only a smallnumber of touch-points or media, or those consisting of a series of verydifferent touch-points or media, are difficult to optimize becauselanguage and images shown to perform better in previous content may notbe applicable to future content, and optimization can only occur betweenone iteration of a campaign and a subsequent iteration with similarcontent and purpose.

Accordingly, several methods intended to maximise the performance ofonline content have been developed, and a selection of these will now bediscussed.

A/B testing (otherwise known as “split testing”) involves the usage oftwo or more mutually exclusive items of content intended to provoke thesame candidate response. The success probability of each content item isbased on the ratio of target action performances to content item usages,and the “winner” is the item with the highest proportion of desiredactions to recipients (i.e. the “response rate”). The winning contentitem is then used in future marketing propositions. Although A/B testingis straightforward to understand and apply, it suffers from a number oflimitations. Chief among these is the trade-off between the number ofcontent parameters to be tested, and the accuracy of the results. Forinstance, if only one content parameter is altered from content item tocontent item, the results clearly indicate the relative performance ofeach selection of that parameter. Exhaustive testing of all combinationsof content parameters of a complex content item one-by-one may requireconsiderable time, and/or a very large number of samples. Conversely,varying combinations of several content parameters from content item tocontent item results in uncertainty regarding the true performanceimpact of each content parameter and selection.

Although more complex than A/B testing, multivariate testing is becomingincreasingly popular and generally takes the form of eitherfractional-factorial testing or full-factorial testing.

Fractional-factorial testing involves the manual or automated selectionof a unique subset of combinations of content items designed to reducethe number of content combinations which need to be tested, and thus thecost of the experiment is reduced. Testing on the subset of combinationsis performed as in the A/B testing methodology outlined above. However,the increased depth of analysis potentially enables greater gains inresponse rate to be achieved. The major limitation of this testingmethod is that interaction effect analysis is restricted, a priori, to aset depth by the experimental design.

Full-factorial multivariate testing involves the generation of a unique,exhaustive set of combinations of content items, and an experimentalmethodology essentially equivalent to the A/B testing methodologyoutlined above. The major drawback of the full-factorial testing methodis that it requires significantly more data for its results to achievestatistical significance.

There remains a need for methods and systems providing improved testingof content performance in relation to recipient response rates.

SUMMARY OF THE INVENTION

According to one aspect of the present invention there is provided acomputer-implemented method for improving recipient response rates tocontent presented via a data communications network to a plurality ofrecipients based upon recipient response data, wherein the contentcomprises a plurality of content factors, each content factor beingselected from one or more corresponding content levels, and wherein themethod comprises the steps of:

a) providing a data structure for containing recipient response dataassociated with content comprising corresponding combinations of contentlevels;

b) generating a plurality of content samples by forming combinations ofcontent levels for a plurality of content factors in accordance withrespective combination weightings derived from recipient response datacontained within the data structure;

c) presenting each one of the plurality of content samples to acorresponding plurality of recipients via the data communicationsnetwork;

d) receiving recipient responses to the content samples via the datacommunications network;

e) analysing the recipient responses to identify one or morecombinations of content levels resulting in high recipient responserates relative to other combinations in accordance with predeterminedstatistical criteria; and

f) storing within the data structure recipient response data associatedwith said combinations of content levels resulting in high recipientresponse rates.

In general, embodiments of the invention store test results (recipientresponse data) within the data structure, and derive weightings forcombinations of content levels, such that these may be used to guidefurther iterations of the testing process based upon incomplete datasets. Advantageously, testing may thus be focused automatically uponthose combinations that emerge as being most likely to result inimproved recipient response rates. Testing is thereby dynamicallyadaptable based upon received recipient responses, and is not restrictedto experimental data sets that have been determined a priori. Thepreferred approach is statistical, whereby in each test stage theprobability that a content sample including a particular combination ofcontent levels will be generated increases with the response rate forthat combination in previous stages.

Preferably, step (e) comprises analysing the recipient responses tocalculate for each one of a plurality of combinations of content levelsa corresponding probability that a recipient response rate to saidcombination of content levels is greater than the average response ratefor all combinations of content levels. Repeating the steps (b) to (f)may be performed so as to further improve recipient response rates. Itis also preferable that the step (b) of generating content samplescomprises a step of ranking the content factors based upon a number ofcontent levels corresponding with each content factor. Step (b) ofgenerating content samples may also comprise forming a full factorial ofcombinations of content levels for the plurality of content factors.

Preferably the data structure comprises a tree structure. This treestructure may comprise: a root node for containing recipient responsedata corresponding with all content samples presented to recipients; andzero or more leaf nodes for containing recipient response datacorresponding with content samples presented to recipients whichcomprise particular content levels and/or combinations of contentlevels.

Preferably, step (b) of generating content samples comprises the stepsof: deriving from the tree structure partial combinations of contentlevels based upon a number of content levels comprising each contentfactor; filtering out partial combinations of content levels based uponthe content levels already selected for the plurality of contentfactors; filtering out partial combinations of content levels for whichmore extended combinations of content levels exist; and selectingpartial combinations of content levels based upon a weighted randomdistribution with the combination weightings as a parameter.

It is a preferred feature of the present invention that the step (e) ofanalysing the recipient responses comprises the steps of: retrievingrecipient response data from the data structure and aggregating saidrecipient response data based upon combinations of content levels; andforming extended combinations of content levels for those combinationsof content levels which have resulted in high recipient response ratesrelative to other combinations. The step of forming extendedcombinations may also comprise generating leaf nodes of the treestructure for containing recipient response data corresponding withcontent samples presented to recipients in a recursive manner for theextended combinations of content levels.

Preferably, each combination weighting (w) is computed based upon aprobability that a recipient response rate to the correspondingcombination of content levels is greater than an average recipientresponse rate for all combinations of content levels (a_(n)) and thenumber of content levels comprising each content factor (k), accordingto the formula:

$w = {\frac{\left( {1 - a_{n}} \right)}{\sum\limits_{i = 0}^{i = k}\;\left( {1 - a_{i}} \right)}.}$

According to a further aspect of the present invention, there isprovided a computer-implemented system for improving recipient responserates to content presented via a data communications network to aplurality of recipients based upon recipient response data, wherein thecontent comprises a plurality of content factors, each content factorbeing selected from one or more corresponding content levels, the systemcomprising:

a) a data structure for containing recipient response data associatedwith content comprising corresponding combinations of content levels;

b) means for generating a plurality of content samples by formingcombinations of content levels for a plurality of content factors inaccordance with respective combination weightings derived from recipientresponse data contained within the data structure;

c) means for presenting each one of the plurality of content samples toa corresponding plurality of recipients via the data communicationsnetwork;

d) means for receiving recipient responses to the content samples viathe data communications network;

e) means for analysing the recipient responses to identify one or morecombinations of content levels resulting in high recipient responserates relative to other combinations in accordance with predeterminedstatistical criteria; and

f) means for storing within the data structure recipient response dataassociated with said combinations of content levels resulting in highrecipient response rates.

According to a still further aspect of the present invention, there isprovided a computer-implemented system of improving recipient responserates to content presented via a data communications network to aplurality of recipients based upon recipient response data, wherein thecontent comprises a plurality of content factors, each content factorbeing selected from one or more corresponding content levels, the systemcomprising one or more computers comprising:

at least one processor;

an interface between said processor and the data communications network;and

one or more computer-readable storage media operatively coupled to theprocessor comprising a data structure for containing recipient responsedata associated with content comprising corresponding combinations ofcontent levels, wherein the storage medium containing programinstructions for execution by the processor, wherein the data structureis contained on at least one storage device and wherein the contentfactors and corresponding one or more content levels are contained on atleast one storage device, said program instructions causing theprocessor to execute the steps of:

a) generating a plurality of content samples by forming combinations ofcontent levels for a plurality of content factors in accordance withrespective combination weightings derived from recipient response datacontained within the data structure;

b) presenting each one of the plurality of content samples to acorresponding plurality of recipients via the data communicationsnetwork;

c) receiving recipient responses to the content samples via the datacommunications network;

d) analysing the recipient responses to identify one or morecombinations of content levels resulting in high recipient responserates relative to other combinations in accordance with predeterminedstatistical criteria; and

f) storing within the data structure recipient response data associatedwith said combinations of content levels resulting in high recipientresponse rates.

According to a still further aspect of the present invention, there isprovided a tangible computer-readable medium having computer-executableinstructions stored thereon for performing a method of improvingrecipient response rates to content presented via a data communicationsnetwork to a plurality of recipients based upon recipient response data,wherein the content comprises a plurality of content factors, eachcontent factor being selected from one or more corresponding contentlevels, the method comprising the steps of:

a) providing a data structure for containing recipient response dataassociated with content comprising corresponding combinations of contentlevels;

b) generating a plurality of content samples by forming combinations ofcontent levels for a plurality of content factors in accordance withrespective combination weightings derived from recipient response datacontained within the data structure;

c) presenting each one of the plurality of content samples to acorresponding plurality of recipients via the data communicationsnetwork;

d) receiving recipient responses to the content samples via the datacommunications network;

e) analysing the recipient responses to identify one or morecombinations of content levels resulting in high recipient responserates relative to other combinations in accordance with predeterminedstatistical criteria; and

f) storing within the data structure recipient response data associatedwith said combinations of content levels resulting in high recipientresponse rates.

Further preferred features and advantages of the invention will beapparent to those skilled in the art from the following description ofpreferred embodiments of the invention, which should not be consideredto be limiting of the scope of the invention as defined in the precedingstatements, or in the claims appended hereto.

BRIEF DESCRIPTION OF THE DRAWINGS

Preferred embodiments of the invention are described with reference tothe accompanying drawings, in which like reference numerals refer tolike features and wherein:

FIG. 1 shows a schematic block diagram of a system for improvingrecipient response rates to content presented via a data communicationsnetwork to a plurality of recipients based upon recipient response data,in accordance with a preferred embodiment of the present invention;

FIG. 2 illustrates the structure of a sample email message, according toan embodiment of the invention;

FIG. 3 is an example of content levels and content factors according toan embodiment of the invention;

FIG. 4 is a flow chart illustrating a preferred method for improvingrecipient response rates to content presented via a data communicationsnetwork to a plurality of recipients based upon recipient response datain accordance with the invention;

FIG. 5 is a flow chart illustrating the steps required to generatecombinations of content factors and content levels in accordance with apreferred embodiment of the invention;

FIG. 6 is a flow chart illustrating the steps required to select acontent level in accordance with a preferred embodiment of theinvention;

FIG. 7 is a flow chart illustrating the steps required to exclude nodesnot matching the current combination of content levels in accordancewith a preferred embodiment of the invention;

FIG. 8 is a flow chart illustrating the steps required to generate apartial combination weighting tree in accordance with a preferredembodiment of the invention;

FIG. 9 is a flow chart illustrating the steps required to recursivelygenerate child nodes in accordance with a preferred embodiment of theinvention;

FIG. 10 is an example subscriber list according to an embodiment of theinvention;

FIG. 11 is an example of a distribution of possible combinations ofcontent levels in a first stage of testing, according to an embodimentof the invention;

FIG. 12 is an example of recipient responses corresponding with thedistribution illustrated in FIG. 11;

FIG. 13 is an example of recipient response data illustrated in FIG. 12aggregated by content level;

FIG. 14 is an example of content level performance data correspondingwith the test data of FIGS. 11 to 13 that is encoded into a treestructure;

FIG. 15 is an example of content levels corresponding with the test dataof FIGS. 11 to 13 ranked in accordance with probability of selection;

FIG. 16 is an example of a distribution of possible combinations ofcontent levels in a second stage of testing, according to an embodimentof the invention;

FIG. 17 is an example of recipient responses corresponding with thedistribution illustrated in FIG. 16;

FIG. 18 is an example of recipient response data illustrated in FIG. 17aggregated by content level;

FIG. 19 is an example of recipient response data illustrated in FIG. 17aggregated by content level pairs; and

FIG. 20 is an example of the content level performance datacorresponding with the test data of FIGS. 16 to 19 that is encoded intoa tree structure.

DESCRIPTION OF PREFERRED EMBODIMENT

A computer-implemented method of improving recipient response rates tocontent presented via a data communications network to a plurality ofrecipients in accordance with a preferred embodiment of the presentinvention is most readily implemented by delivery of content torecipients via email and it will therefore be convenient to describe theinvention in that environment.

It should be understood that the invention is not limited to thispreferred embodiment. In an alternative embodiment of the presentinvention the delivery of content to recipients may be facilitated byweb servers which allow the user of a terminal to access a dynamicweb-based service. A service of this nature would facilitate thedelivery of content to users, and allow for the calculation of responserates to said content. Advantageously, the content would be delivered tousers by accessing a webpage at a predefined Uniform Resource Locator(URL).

FIG. 1 illustrates an exemplary system 100 in which preferredembodiments of the invention may be implemented. The system includes aserver 102 and at least one user terminal 104, both of which areconnected to a network 106, which may be, for example, the Internet.Also connected to the network 106 are a plurality of user terminalsand/or servers, eg 108, 110. It will be appreciated that FIG. 1 depictsthe system 100 schematically only, and is not intended to limit thetechnology employed in the servers, user terminals and/or communicationslinks. The user terminals in particular may be wired or wirelessdevices, and their connections to the network may utilize varioustechnologies and bandwidths. For example, applicable user terminalsinclude (without limitation): PC's with wired (eg LAN, cable, ADSL,dial-up) or wireless (eg WLAN, cellular) connections; and wirelessportable/handheld devices such as PDA's or mobile/cellular telephones.The protocols and interfaces between the user terminals and the serversmay also vary according to available technologies, and include (againwithout limitation): wired TCP/IP (Internet) protocols; GPRS, WAP and/or3G protocols (for handheld/cellular devices); Short Message Service(SMS) messaging for digital mobile/cellular devices; and/or proprietarycommunications protocols.

The server 102 includes at least one processor 112 as well as a database114, which would typically be stored on a secondary storage device ofthe server 102, such as one or more hard disk drives. Server 102 furtherincludes at least one storage medium 116, typically being a suitabletype of memory, such as random access memory, for containing programinstructions and transient data related to the operation of the searchengine as well as other necessary functions of the server 102. Inparticular, the memory 116 contains a body of program instructions 118implementing the method and system in accordance with preferredembodiments of the invention. The body of program instructions 118includes instructions for improving recipient response rates to contentpresented via a data communications network, such as a web-basedinterface, to a plurality of recipients based upon recipient responsedata, the operation of which will be described hereafter.

It should be appreciated that the hardware used to implement the methodof the invention may be conventional in nature or specifically designedfor the purpose. The hardware structure shown in FIG. 1 is merely onepossible embodiment and any other suitable structure may be utilised.

Preparation

FIG. 2 illustrates the structure of a sample email 200, representingcontent presented via a data communications network to a plurality ofrecipients in accordance with a preferred embodiment of the invention.Every email 200 contains two distinct sections:

1. a set of “headers” 202, which generally contain metadata such as thesender of an email, the time at which it was sent, the Internet Protocol(IP) address of the originating computer, the subject of the email etc;and

2. a “body” 204, which includes the content of the email message,expressed, for example, in plain text or as HTML.

A preferred embodiment of the present invention requires furthersubdivision of the email header 202 and body 204. For example, FIG. 2illustrates the subdivision of the body into “Title” 206, “Image” 208and “Footer” 210, whereas the header contains a “Subject” 212. Each ofthese subdivisions is modelled as a content factor, which contains oneor more mutually exclusive content levels representing the alternativesavailable for the content factor. For example, the content levels forthe “Image” content factor represent the various images which areavailable for use in the email.

The number of content factors and content levels is arbitrary, andfreely selectable by the user in accordance with the logical structureof the email message itself. An example table 300 illustrating contentlevels and content factors is shown in FIG. 3. In this example, thecontent factors illustrated are “Subject”, “Title”, “Image” and“Footer”. Corresponding to each of these content factors is at least onecontent level. The “Subject” content levels 302 shown in FIG. 3 are“Subject 1”, “Subject 2” and “Subject 3”, the “Title” content levels 304shown are “Title 1” and “Title 2”, the “Image” content levels 306 shownare “Image 1”, “Image 2” and “Image 3”, and the “Footer” content level308 shown is “Footer 1”.

FIGS. 4 to 9 are flow charts illustrating the steps in a methodembodying the present invention. Briefly, FIG. 4 is a flow chart 400which illustrates a preferred method for improving recipient responserates to content presented via a data communications network to aplurality of recipients based upon recipient response data, inaccordance with the present invention. FIG. 5 is a flowchart 500 whichillustrates the steps required by § 1.3 in flow chart 400. FIG. 6 is aflowchart 600 which illustrates the steps required by § 2.5 in flowchart 500. FIG. 7 is a flow chart 700 which illustrates the stepsrequired by § 3.3 in flow chart 600. FIG. 8 is a flow chart 800 whichillustrates the steps required by § 1.9 of flow chart 400. FIG. 9 is aflow chart 900 which illustrates the steps required by § 5.3 of flowchart 800. Further details will now be described with reference to theflow charts 400-900.

Broadcast: Stage 1

In accordance with the general method shown in flow chart 400, at § 1.1an initial request is received to initiate an optimisation process basedon the results of an email broadcast. At § 1.2, a tree structure isinitialized containing nodes which represent the content factors (forexample “Subject”, “Title”, “Image” and “Footer”) to be tested. Adetermination is made as to the number of recipients (n) who should formpart of each stage of the optimisation procedure based upon the desiredstatistical level of confidence of the results. In the first stage ofthe broadcast, a subset of the potential recipients are selected basedon an initial estimate of the sample size required for reliableanalysis. For an exemplary recipient database, an initial sample size of10,000 is appropriate. Preferably, these recipients are selected from apool of email subscribers. The following step, at § 1.3, requires thegeneration of n combinations of content levels to be delivered to thefirst n recipients, which make up the first stage of the optimisationprocess. Combinations of content levels are generated at random, withone content level (eg, one of “Subject 1”, “Subject 2” or “Subject 3”)being selected for each content factor (eg “Subject”).

After the email structure is defined, and content factors and contentlevels are chosen (in accordance with § 1.1 and 1.2 of flow chart 400),then at § 1.4, the generated content is delivered to the n recipients inthe form of an email message. The recipients' data is stored in adatabase as a series of records, each with an email address (or otherunique identifier determining the destination of the message), andassociated subscriber information (such as name, geographic location, orage). An example subscriber list 1000 is illustrated in FIG. 10. In thisexample, the information stored in the subscriber list includes “EmailAddress” 1002, “Age” 1004 and “Postal Code” 1006, although the actualinformation stored will depend largely on the demographics of thesubscribers and the required outcomes of the optimisation process.

Given a sufficiently large sample size, FIG. 11 illustrates thedistribution of the possible combinations of content levels 1100. Inthis example the total number of possible combinations of content levels(eg all combinations of the “Subject”, “Title”, “Image” and “Footer”content levels) is 18. The distribution of each of these 18 combinationsin the first stage is performed at random in accordance with a uniformprobability distribution function, and the proportion of recipients 1102to which each combination is sent is therefore approximately 1/18=5.56%.After an email message containing a combination of content levels issent to a recipient, details of the combination are stored in a databaseand linked to the recipient's unique identifier.

Preferably, the delivery of content to recipients and the tracking ofrecipient responses, shown at § 1.13 and 1.14 in flow chart 400, isperformed via an external system such as the SMTP or SOAP protocols.During this process, event data (such as email opens, newslettersubscriptions, product purchases, page views etc) are collected andaggregated, as indicated at § 1.5 of flow chart 400, based upon thecontent level combination sent to the recipient responsible forgenerating the event.

Analysis: Stage 1

As the first stage of the broadcast is sent, the unique identifier ofrecipients “responding” (i.e. performing a certain desired action) isrecorded, in accordance with § 1.5 of flow chart 400. At § 1.6, adetermination is made as to whether there is recipient response datawhich has not yet been analysed, and to ensure that an email message hasbeen sent to all n recipients selected at the beginning of the stage. Ifthere are no remaining recipients to process then the optimisationprocess is complete, § 1.15. Otherwise, a determination is made as towhether there is sufficient data available to begin the analysis phaseof the process. An example of the output of this step is illustrated inFIG. 12. This example illustrates the recipient responses 1200corresponding to the distribution in the first stage of the process. Inparticular, it shows the numbers of recipients 1202 that received aparticular combination of content levels, and the number of recipientresponses 1204 to a particular combination of content levels.

Because the number of responses is generally low, it is impossible todetermine the best-performing combination of content levels with anydegree of statistical confidence. Thus, in accordance with § 1.9 and1.11 of flow chart 400, the algorithm aggregates recipient and responsecounts by content levels, yielding the results 1300 shown in FIG. 13.This example shows the response rate 1306 in stage 1 to each of thecontent levels, which represents the number of recipient responses 1304to a particular content level as a percentage of the total number 1302of recipients that received an email message containing that contentlevel. For example, the results 1300 indicate that 3320 recipientsreceived an email message containing the “Subject 1” content level, andonly 105 of these recipients responded. Therefore, the response rate tothe “Image 1” content level is 3.16%. In the results shown in FIG. 13,it is indicated that the response rate of the “Title 1” content level is3.47% and the response rate of the “Image 3” content level is 3.49%.These are the two highest performing content levels, with respect to theaverage response rate of 3.14%.

The response rate information for the high performing content levels isthen used to update the content tree structure, in accordance with § 1.9of flow chart 400. The tree structure encodes the top-performingcombinations of content levels, such that child nodes have higherresponse rates than their parents, as shown in FIG. 14. In the examplestructure 1400, the “Path to Tree Nodes” 1402 are listed with theircorresponding recipient response rate 1404. The “<root>” node stores theoverall average performance of all email messages in the broadcast,whereas for example, the “<root>/Title 1” node stores the performance ofall email messages in the broadcast containing the “Title 1” contentlevel. A more complex example is “<root>/Title 1/Image 3/Subject 1” (notshown), which represents a unique combination of “Subject”, “Title” and“Image” content levels within the broadcast. As shown in FIG. 11, the“Footer 1” content level is used in every combination of content levels,as it is the only available content level for the “Footer” contentfactor, and therefore there is no requirement that this be encoded inthe tree.

Considering flow chart 800, § 5.1 to 5.4 are the steps required by § 1.9for generating a partial combination weighting tree for a stage of theoptimisation process. At § 5.2, the response data generated byrecipients in the stages since the previous analysis phase is aggregatedby type (eg the particular response event) and the particularcombination of content levels which produced the response. Then at §5.3, child nodes are recursively generated, starting from the root node.Considering flow chart 900, § 6.1 to 6.5 are the steps required by § 5.3for recursively generating child nodes. In steps § 6.2 to 6.3 of flowchart 900, the system generates tree nodes in a recursive fashion forevery combination of content levels in distinct factors, provided thatthe given combination of content levels was received by more than acertain number of recipients (eg 1000 recipients, although thisthreshold is selected based on expected response rates and the desiredconfidence level), and the combination of content levels is no morespecific than needed to identify a distinct set of linked levels (eg ifcertain content levels are required by external metadata to be sent toparticular types of recipients). Each node contains the conditionalprobability of an event occurring per recipient for the content levelthat the node is concerned with, given that the recipient has alsoreceived each content level identified by the tree path from the rootnode to the node in question. At § 6.4, the extension of the treestructure is returned, and then at § 5.4 the entire length of the treestructure from the child nodes down to the root node is returned andstored in the database.

If it is determined, at § 1.7, that the data is not statisticallysignificant then the number of recipients per stage (n) is increased, inaccordance with § 1.8 and 1.10 in flow chart 400, the contentcombination weightings are updated in the tree structure (in accordancewith § 1.11) and the process of generating and delivering content (ie §1.3 to 1.6) is repeated until there is sufficient data to perform ananalysis.

Broadcast: Stage 2

If the results of “Analysis: Stage 1” are inconclusive, the algorithmincreases the sample size of future stages by a fixed amount, selectableby the user (in accordance with § 1.8 of flow chart 400). Otherwise, thesample size remains the same as in Stage 1, and the broadcast proceedsas outlined above in “Broadcast: Stage 1”, except that the previouslyrandom composition of the email messages sent is modified to take intoconsideration the data stored in the tree structure, as shown in FIG. 14(in accordance with § 1.10 of flow chart 400).

Considering the method shown in flow chart 500, § 2.1 to 2.7 are thesteps performed in generating the n combinations of content levels, inaccordance with § 1.3 of flow chart 400. Each email message, in thesubsequent stages of the optimisation process, is generatedfactor-by-factor, starting with the content factor for which the fewestcontent levels are defined. The operation of the content level selectionalgorithm is described below.

At § 2.2 of flow chart 500, all available content levels are retrievedfrom the database.

Then at § 2.3, and beginning with the content factor with the fewestcontent levels (in the example, the “Title” content factor has thefewest number of content levels available), a ranked list of all contentlevels for that content factor are generated as follows:

-   -   (i) In accordance with decision § 2.4, if there is more than one        content level for the current content factor (§ 2.4), then the        most specific content level performance tree nodes (as shown in        FIG. 14) applicable to the current email message is retrieved in        accordance with § 2.5 of flow chart 500. Considering flow chart        600, § 3.1 to 3.6 are the steps required by § 2.5 for selecting        a level for the current content factor. At § 3.2, all nodes in        the partial combination weighting tree structure are retrieved.        Then at § 3.3, it is necessary to exclude nodes not matching the        current combination of content levels selected. Considering flow        chart 700, § 4.1 to 4.6 or 4.7, are the steps required by § 3.3        to exclude nodes that do not match the current combination of        content levels.    -   At § 4.2, it is necessary to determine the content factor for        each content level associated with the current node. In        accordance with decision § 4.3, if there are no content levels        for any of these content factors present in the current        combination, then specificity is determined by the number of        levels in the “Path to Tree Node” that match the set of content        levels already selected for inclusion in the email message, in        accordance with § 4.4 and 4.5 of flow chart 500. If a node        contains a content level that contradicts a selection already        made for the current email message (because the respective        content factor for that content level has already been processed        for the current email message, and a different content level        selected), the node is removed from consideration at § 4.6.        Otherwise, the node is included in the result set at § 4.7. In        the example, content level “Title 1” would be selected as the        content level for the “Title” content factor, and “Image 3”        would be selected as the content level for the “Image” content        factor. No content level would be selected for the “Subject”        content factor, as no content levels are present for this        content factor in the level performance tree.    -   (ii) At § 3.4 of flow chart 600, the list of tree nodes is        sorted by descending response rate, and the content level        specified for the current content factor is extracted from each        of the retrieved tree nodes and returned at § 3.6. In accordance        with § 3.5 of flow chart 600, this selection is based upon a        weighted random distribution with the node response probability        as a parameter.    -   (iii) Any content levels for the current content factor        available but not already retrieved are appended to the list.        Given the example selections outlined above at (i), content        level “Title 2” would be appended to the list for the “Title”        content factor, content levels “Image 1” and “Image 2” would be        appended to the list for the “Image” content factor, and content        levels “Subject 1”, “Subject 2” and “Subject 3” would be        appended to the list for the “Subject” content factor.

A content level is selected from the ranked list of content levels (inaccordance with § 2.5 of flow chart 500), and inserted into the emailmessage in the location defined for the current content factor, inaccordance with § 2.6 and 2.7 of flow chart 500. The content levelselected is determined statistically, with each content level having aprobability of selection defined by a weighting function taking asparameters the rank of the content level, and the recipient count andresponse rate stored in the relevant tree node. In the exemplaryimplementation, the weighting function is

$w = \frac{\left( {1 - a_{n}} \right)}{\sum\limits_{i = 0}^{i = k}\;\left( {1 - a_{i}} \right)}$where a_(n) is the probability that the response rate of content level nis greater than the average response rate (based on standard statisticalconfidence tests, for example assuming a normal distribution) or 0.5 inthe case of a content level not extracted from the performance tree, andk is the number of content levels in the ranked list 1502. An example ofthe list generated for the “Image” content factor 1500 in Steps 1 and 2above, along with the probability of selection 1504 of each contentlevel as determined by the above weighting function is illustrated inFIG. 15. In this example, the “Image 3” content level is ranked first asit has a probability of selection of 0.44, which is greater than that ofthe “Image 1” and “Image 2” content levels, which both have aprobability of selection of 0.28.

Steps § 2.3 through 2.7 above are repeated until a content level hasbeen selected for each content factor defined for the current emailmessage (in accordance with § 1.3 of flow chart 400). Once an emailmessage has been generated, it is sent to the designated recipient inaccordance with § 1.4 of flow chart 400. Based on the example data, thedistribution of combinations of content levels in a Stage 2 broadcast isillustrated in FIG. 16. The example 1600 shows the proportion ofrecipients to which combinations of content levels will be distributed,as with FIG. 11, although unlike Stage 1, the distributions in Stage 2are no longer uniform due to the adjusted combination weightings. Allcombinations of content levels containing the “Image 3” and “Title 1”content levels (ie “Subject 1, Title 1, Image 3, Footer 1”, “Subject 2,Title 1, Image 3, Footer 1” and “Subject 3, Title 1, Image 3, Footer 1”)are distributed to 9.24% of recipients, which is the highest proportionof the possible combinations. This is because the “Image 3” and Title 1”content levels are the highest-performing content levels.

Analysis: Stage 2

The analysis at Stage 2 proceeds as described in “Analysis: Stage 1”above. Considering that the distribution of email messages is weightedtowards combinations including high-performing content levels, exemplaryresults 1700 of the first analysis step of Stage 2 are shown in FIG. 17.This example is generated by repeating the analysis, the results 1200 ofwhich are shown in FIG. 12, and shows the recipient responses 1700corresponding to the distribution in the first stage of the process. Inparticular, the example shows the numbers of recipients 1702 thatreceived a particular combination of content levels, and the number ofrecipient responses 1704 to a particular combination of content levelsin Stage 2 of the process. The algorithm then aggregates recipientresponse data by content level, as previously in “Analysis: Stage 1”,the results 1300 of which are illustrated in FIG. 13. The correspondingresults 1800 of aggregating the recipient response data shown in FIG. 17are illustrated in FIG. 18. This example shows the response rate 1806 instage 2 to each of the content levels, which represents the number ofrecipient responses 1804 to a particular content level as a percentageof the total number of recipients that received an email messagecontaining that content level 1802.

In addition, the non-uniform distribution of recipient response data(weighted towards combinations including successful content levels)enables deeper analysis of higher-performing combinations of contentlevels. In addition to analysis by content level, there may be enoughdata for analysis by pairs of content levels. The results 1900 ofaggregating the recipient response data by pairs of content levels areillustrated in FIG. 19. Based on the recipient response data in FIG. 19,the average response rate is 3.30%. Content level pair “Image 3, Title1” has a response rate of 4.12%, and content level pair “Subject 3,Image 3” has a response rate of 3.93%. Only these two content levelpairs (“Image 3, Title 1” and “Subject 3, Image 3”) have recipientresponse rates exceeding the average response rate by a significantmargin. The margin may be predetermined by the user and can be increasedor decreased depending on the results obtained during the optimisationprocess. The content level performance tree structure 2000 for this dataset, generated as in Stage 1, is illustrated in FIG. 20. The treestructure 2000 encodes the top-performing combinations of content levelsand content level pairs, such that child nodes have higher responserates than their parents. In the example shown in FIG. 20, the “Path toTree Nodes” 2002 are listed with their corresponding recipient responserate 2004. The example indicates that the “<root>/Image 3/Title 1” nodestores the performance (ie 4.12% response rate) of all email messages inthe broadcast containing the “Image 3, Title 1” content level pair.Similarly, the “<root>/Image 3/Subject 3” node stores the performance(ie 3.93% response rate) of all email messages in the broadcastcontaining the “Image 3, Subject 3” content level pair.

Broadcast & Analysis: Subsequent Stages

The broadcast and analysis algorithm operates as described in Stage 2above for each subsequent stage, until an email message has beendelivered to each recipient within the customer database. If there areno longer any recipients left to process (§ 1.6 of flow chart 400, aspreviously discussed) then the optimisation process ends (§ 1.15).

At each stage, high-performing combinations of content levels areidentified with greater certainty, and a larger proportion of messagesare generated which include such combinations. Accordingly, improvementsin recipient response rates may be achieved by ongoing dynamicadaptation of messages which focuses upon combinations found to bestatistically more likely to produce desired responses.

It will be understood that while preferred embodiments of the inventionhave been described herein, these should not be considered to limit thescope of the invention, which is defined by the claims appended hereto.

I claim:
 1. A method for improving recipient response rates to SMSmessage or email content transmitted via a data communications networkto a plurality of recipients, the method comprising the steps of: a)storing in one or more computer-readable data storage media contentcomprising a plurality of content levels; b) storing in the one or morecomputer-readable data storage media a weighting value for each of thecontent levels; c) generating by at least one computer a plurality ofSMS message or email content samples by forming combinations of contentlevels for a plurality of content factors, the SMS message or emailcontent samples being configured for inducing a recipient to generate anevent representing a response; d) for each SMS message or email contentsample of the plurality of SMS message or email content samples, the atleast one computer calculating a combination weighting value from thestored weighting values corresponding with the content levels comprisingthe SMS message or email content sample; e) selecting, by the at leastone computer, a subset of the plurality of recipients and transmitting,by the at least one computer, to each member of the subset only one ofthe plurality of SMS message or email content samples, the one SMSmessage or email content sample being selected according to thecombination weighting values; f) receiving, by the at least onecomputer, one or more recipient responses via the data communicationsnetwork from one or more recipients of the subset, wherein eachrecipient response is a request according to one or more of the groupconsisting of: HTTP, SMTP, and SOAP; g) analyzing, by the at least onecomputer, the one or more recipient responses by computing for each SMSmessage or email content sample a response rate, wherein each responserate is a ratio between a number of recipients of the corresponding SMSmessage or email content sample and a number of recipient responses tothe corresponding content sample to identify one or more combinations ofcontent levels resulting in high recipient response rates relative toother combinations in accordance with predetermined statisticalcriteria; h) adjusting, by the at least one computer, the weightingvalues in the one or more data storage media such that the, or each,combination of content levels resulting in high recipient response ratehas an increased probability of subsequent selection, and storing theadjusted weighting values within the one or more data storage media; andi) repeating at least steps d) and e) utilizing the adjusted weightingvalues to increase the probability of receiving recipient responses. 2.The method of claim 1 wherein the step (g) comprises analyzing therecipient responses to calculate for each one of a plurality ofcombinations of content levels a corresponding probability that arecipient response rate to said combination of content levels is greaterthan the average response rate for all combinations of content levels.3. The method of claim 1 which includes repeating the steps d) to h) soas to further improve recipient response rates.
 4. The method of claim 1wherein the step (c) of generating content samples comprises a step ofranking the content factors based upon a number of content levelscorresponding with each content factor.
 5. The method of claim 1 whereinthe step (c) of generating content samples comprises forming a fullfactorial of combinations of content levels for the plurality of contentfactors.
 6. The method of claim 1 comprising storing the recipientresponse data and the weighting values in a data structure stored in theone or more storage media, wherein the data structure comprises a treestructure.
 7. The method of claim 6 wherein the tree structurecomprises: a root node for containing recipient response datacorresponding with all content samples presented to recipients; and zeroor more leaf nodes for containing recipient response data correspondingwith content samples presented to recipients which comprise particularcontent levels and/or combinations of content levels.
 8. The method ofclaim 7 wherein the step (c) of generating content samples comprises thesteps of: deriving from the tree structure partial combinations ofcontent levels based upon a number of content levels comprising eachcontent factor; filtering out partial combinations of content levelsbased upon the content levels already selected for the plurality ofcontent factors; filtering out partial combinations of content levelsfor which more extended combinations of content levels exist; andselecting partial combinations of content levels based upon a weightedrandom distribution with the combination weightings as a parameter. 9.The method of claim 1 wherein the step (g) of analyzing the recipientresponses comprises the steps of: retrieving recipient response datafrom the data structure and aggregating said recipient response databased upon combinations of content levels; and forming extendedcombinations of content levels for those combinations of content levelswhich have resulted in high recipient response rates relative to othercombinations.
 10. The method of claim 7 wherein the step (g) ofanalyzing the recipient responses comprises the steps of: retrievingrecipient response data from the data structure and aggregating saidrecipient response data based upon combinations of content levels;forming extended combinations of content levels for those combinationsof content levels which have resulted in high recipient response ratesrelative to other combinations; and generating leaf nodes of said treestructure for containing recipient response data corresponding withcontent samples presented to recipients in a recursive manner for saidextended combinations of content levels.
 11. The method of claim 1wherein each combination weighting value (w) is computed based upon aprobability that a recipient response rate to the correspondingcombination of content levels is greater than an average recipientresponse rate for all combinations of content levels (a_(n)) and thenumber of content levels comprising each content factor (k), accordingto the formula:$w = {\frac{\left( {1 - a_{n}} \right)}{\sum\limits_{i = 0}^{i = k}\;\left( {1 - a_{i}} \right)}.}$12. A non-transitory computer-readable medium having computer-executableinstructions stored thereon for performing a method for improvingrecipient response rates to SMS message or email content transmitted viaa data communications network to a plurality of recipients, the methodcomprising the steps of: a) storing in one or more computer-readabledata storage media content comprising a plurality of content levels; b)storing in the one or more computer-readable data storage media aweighting value for each of the content levels; c) generating aplurality of SMS message or email content samples by formingcombinations of content levels for a plurality of content factors, theSMS message or email content samples being configured for inducing arecipient to generate an event representing a response; d) for each SMSmessage or email content sample of the plurality of SMS message or emailcontent samples, calculating a combination weighting value from thestored weighting values corresponding with the content levels comprisingthe SMS message or email content sample; e) selecting a subset of theplurality of recipients and transmitting to each member of the subsetonly one of the plurality of SMS message or email content samples, theone SMS message or email content sample being selected according tocombination weighting values; f) receiving one or more recipientresponses via the data communications network from one or morerecipients of the subset, wherein each recipient response is a requestaccording to one or more of the group consisting of: HTTP, SMTP, andSOAP; g) analyzing the one or more recipient responses by computing foreach SMS message or email content sample a response rate, wherein eachresponse rate is a ratio between a number of recipients of thecorresponding SMS message or email content sample and a number ofrecipient responses to the corresponding content sample to identify oneor more combinations of content levels resulting in high recipientresponse rates relative to other combinations in accordance withpredetermined statistical criteria; h) adjusting the weighting values inthe one or more data storage media such that the, or each, combinationof content levels resulting in high recipient response rate has anincreased probability of subsequent selection, and storing the adjustedweighting values within the one or more data storage media; and i)repeating at least steps d) and e) utilizing the adjusted weightingvalues to increase the probability of receiving recipient responses. 13.A system for improving recipient response rates to SMS message or emailcontent transmitted via a data communications network to a plurality ofrecipients, the system comprising at least one computer and one or morecomputer-readable storage media, wherein: a) the one or more storagemedia includes a data structure in which can be stored recipientresponse data associated with content comprising correspondingcombinations of content levels; b) the at least one computer isconfigured to store in the data structure in the one or more storagemedia a weighting value for each of the content levels; c) the at leastone computer is configured to generate a plurality of SMS message oremail content samples by forming combinations of content levels for aplurality of content factors, the SMS message or email content samplesbeing configured for inducing a recipient to generate an eventrepresenting a response; d) the at least one computer is configured tocalculate, for each SMS message or email content sample of the pluralityof SMS message or email content samples, a combination weighting valuefrom the stored weighting values corresponding with the content levelscomprising the SMS message or email content sample; e) the at least onecomputer is configured to select a subset of the plurality of recipientsand transmit to each member of the subset only one of the plurality ofSMS message or email content samples, the one SMS message or emailcontent sample being selected according to the combination weightingvalues; f) the at least one computer is configured to receive one ormore recipient responses via the data communications network from one ormore recipients of the subset, wherein each recipient response is arequest according to one or more of the group consisting of: HTTP, SMTP,and SOAP; g) the at least one computer is configured to analyze therecipient responses by computing for each SMS message or email contentsample a response rate, wherein each response rate is a ratio between anumber of recipients of the corresponding SMS message or email contentsample and a number of recipient responses to the corresponding contentsample, to identify one or more combinations of content levels resultingin high recipient response rates relative to other combinations inaccordance with predetermined statistical criteria; h) the at least onecomputer is configured to adjust the weighting values in the datastructure in the one or more data storage media such that the, or each,combination of content levels resulting in high recipient response ratehas an increased probability of subsequent selection, and store theadjusted weighting values within the data structure in the one or morestorage media, to increase the probability of receiving recipientresponses upon subsequent repetition of at least steps d) and e).
 14. Acomputer-implemented system for improving recipient response rates toSMS message or email content transmitted via a data communicationsnetwork to a plurality of recipients, the system comprising one or morecomputers comprising: at least one processor; an interface between saidprocessor and the data communications network; and one or morecomputer-readable storage media operatively coupled to the processorcomprising a data structure for containing recipient response dataassociated with content comprising corresponding combinations of contentlevels, and a weighting value associated with each one of the contentlevels, wherein the storage media further comprises program instructionsfor execution by the processor, said program instructions causing theprocessor to execute the steps of: a) generating a plurality of SMSmessage or email content samples by forming combinations of contentlevels for a plurality of content factors in accordance with respectivecombination weightings derived from recipient response data containedwithin the data structure; b) for each SMS message or email contentsample of the plurality of SMS message or email content samples, the atleast one computer calculating a combination weighting value from thestored weighting values corresponding with the content levels comprisingthe SMS message or email content sample; c) selecting a subset of theplurality of recipients and transmitting to each member of the subsetonly one of the plurality of SMS message or email content samples, theone SMS message or email sample being selected according to thecombination weighting values; d) receiving one or more recipientresponses via the data communications network from one or morerecipients of the subset, wherein each recipient response is a requestaccording to one or more of the group consisting of: HTTP, SMTP, andSOAP; e) analyzing the recipient responses by computing for each SMSmessage or email content sample a response rate, wherein each responserate is a ratio between a number of recipients of the corresponding SMSmessage or email content sample and a number of recipient responses tothe corresponding content sample to identify one or more combinations ofcontent levels resulting in high recipient response rates relative toother combinations in accordance with predetermined statisticalcriteria; f) adjusting the weighting values in the one or more datastorage media such that the, or each, combination of content levelsresulting in high recipient response rate has an increased probabilityof subsequent selection, and storing the adjusted weighting valueswithin the data structure in the one or more storage media; and g)repeating at least steps b) and c) utilizing the adjusted weightingvalues to increase the probability of receiving recipient responses. 15.The system of claim 14 wherein the data structure comprises a treestructure.
 16. The system of claim 15 wherein the tree structurecomprises: a root node for containing recipient response datacorresponding with all content samples presented to recipients; and zeroor more leaf nodes for containing recipient response data correspondingwith content samples presented to recipients which comprise particularcontent levels and/or combinations of content levels.
 17. The system ofclaim 16 wherein said program instructions cause the processor, in thestep (a) of generating content samples, to execute the steps of:deriving from the tree structure partial combinations of content levelsbased upon a number of content levels comprising each content factor;filtering out partial combinations of content levels based upon thecontent levels already selected for the plurality of content factors;filtering out partial combinations of content levels for which moreextended combinations on content levels exist; and selecting partialcombinations of content levels based upon a weighted random distributionwith the combination weightings as a parameter.
 18. The system of claim14 wherein said program instructions cause the processor, in the step(d) of analysing the recipient responses, to execute the steps of:aggregating candidate response data based upon combinations of contentlevels and response type; generating extended combinations of contentlevels based upon the number of content levels available to a candidateand the probability of positive candidate responses to a contentcombination.
 19. The system of claim 14 wherein said programinstructions cause the processor to compute each combination weighting(w) based upon a probability that a recipient response rate to thecorresponding combination of content levels is greater than an averagerecipient response rate for all combinations of content levels (a_(n))and the number of content levels comprising each content factor (k),according to the formula:$w = {\frac{\left( {1 - a_{n}} \right)}{\sum\limits_{i = 0}^{i = k}\;\left( {1 - a_{i}} \right)}.}$20. The computer-implemented system of claim 14, wherein the programinstructions comprise program instructions causing the computer toexecute the steps of: subsequent to step e) and prior to step g),determining according to one or more predetermined statistical criteriathat the analyzing in step g) has failed to yield statisticallysignificant data; and consequent to the determining, using in step g) asubset of the plurality of recipients that is relatively larger than thesubset used in the prior step c).
 21. The tangible computer-readablemedium of claim 12, wherein the method comprises: subsequent to step g)and prior to step i), determining according to one or more predeterminedstatistical criteria that the analyzing in step g) has failed to yieldstatistically significant data; and consequent to the determining, usingin step i) a subset of the plurality of recipients that is relativelylarger than the subset used in the prior step e).
 22. The method ofclaim 1, comprising: subsequent to step g) and prior to step i),determining according to one or more predetermined statistical criteriathat the analyzing in step g) has failed to yield statisticallysignificant data; and consequent to the determining, using in step i) asubset of the plurality of recipients that is relatively larger than thesubset used in the prior step e).